Enhancing low quality images of naturally illuminated scenes

ABSTRACT

A method enhances images of a naturally illuminated scene. First, a set of images I i (x, y) is acquired of a scene. Each image is acquired under a different uncontrolled illumination. For each image I i (x, y), intensity gradients ∇I i (x, y) are determined, and each image is weighted with weights according to the intensity gradients. The weighted images are then combined to construct an enhanced image I′.

FIELD OF THE INVENTION

The invention relates generally to rendering non-photorealistic images,and more particularly to generating non-photorealistic images fromimages acquired of natural scenes.

BACKGROUND OF THE INVENTION

Conventional cameras are excellent for capturing realism in scenes.However, conventional images are frequently insufficient for manyapplications where instead of physical realism, altered images aredesired. The alteration can enhance some details, while de-emphasizingother details.

Therefore, many modern digital cameras perform image processing, such asnon-linear color mapping, edge sharpening, or saturating blue shades inimages of outdoor scenes to make water or sky look more lively, whilestill maintaining near-photorealistic appearance.

Similarly, a photographer can illuminate a scene with soft, side or backlighting to selectively add contrast to the scene. In addition, imagescan be retouched using a photo-editor that applies various image filtersand enhancers.

Besides being aesthetically pleasing, enhanced and reduced complexityimages are also useful for highlighting or clarifying selected featuresin technical illustrations, or high-quality thumbnail images. With a fewexceptions, available techniques for image enhancement, imageabstraction or image stylization involve capturing and processing asingle image as the input, see DeCarlo et al., “Stylization andAbstraction of Photographs,” Proceedings of Siggraph '02, ACM Press,2002, and Hertzmann, “Painterly Rendering with Curved Brush Strokes ofMultiple Sizes,” Proceedings of Siggraph '98, ACM Press, 1998.

Even with state-of-the-art passive and active image processingtechniques, it is still difficult to acquire accurate depth, normal andshape information. It is also difficult to generate novel views of ascene from acquired images.

In computer graphics, one approach hides these problems by using highquality images. Texture mapping can ‘hide’ low quality in range images,and light fields enable data-dense views instead of relying on datarepresentation.

At the same time, digital cameras continue to evolve at a rapid pace.Advancements include higher pixel resolutions, more bits per pixel toincrease the dynamic range, higher frame rates, and the ability to takemultiple pictures with different lengths of exposure and wavelength,e.g., infrared.

Non-photorealistic images (NPR) intentionally appear different fromphotographs. NPR images can be classified broadly as artistic orutilitarian, e.g., technical illustrations. NPR images can emphasizeimportant features such as object edges. Moving parts can be shown indifferent colors, and less important details such as shadows and other‘clutter’ can be reduced or eliminated. Furthermore, NPR images candecouple image resolution from contained information.

The input for stylized image generation can be 3D geometricrepresentations or images, see Markosian et al., “Real-TimeNonphotorealistic Rendering,” Proceedings of Siggraph '97, Whitted, Ed.,Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH,pp. 415–420, 1997, Gooch et al., “Using Non-Photorealistic Rendering toCommunicate Shape,” Siggraph '99 Course Notes, Course onNon-Photorealistic Rendering, Green, Ed. Ch. 8., 1999, Hertzmann,“Introduction to 3D Non-Photorealistic Rendering: Silhouettes andOutlines,” Siggraph '99 Course Notes on Non-Photorealistic Rendering,Green, Ed. New York, Ch. 7, 1999, and Kowalski, “Art-Based Rendering ofFur, Grass, and Trees,” Proceedings of Siggraph '99, Computer GraphicsProceedings, Annual Conference Series, ACM SIGGRAPH, pp. 433–438, 1999.

Prior art techniques for generating stylized images from a single imagehave involved morphological operations, image segmentation, edgedetection and color assignment. However, those techniques are limited bytheir dependence on a single input image. Some of techniques aim forstylized depiction, see Ostromoukhov, “Digital facial engraving,”Proceedings of Siggraph '99, Rockwood, Ed., ACM SIGGRAPH, pp. 417–424,1999, while others try to enhance legibility. Interactive techniquessuch as rotoscoping are effective as well.

Methods to combine information from multiple images into one have beenexplored for various other applications. They vary from tone mapping forcompression of variable-exposure high-dynamic range images, see Fattalet al., “Gradient Domain High Dynamic Range Compression,” Proceedings ofSigggraph '02, ACM SIGGRAPH, 2002, and Reinhard et al., “PhotographicTone Reproduction for Images,” Proceedings of Siggraph '02, ACMSIGGRAPH, 2002.

Some techniques consider changing atmospheric conditions to extract 3Dinformation and perform fog elimination, see Nayar et al., “High dynamicrange imaging: Spatially varying pixel exposures,” IEEE CVPR, 2000.Active illumination methods have been used for depth extraction andphotometric stereo. Unfortunately, active illumination is unstable atdepth discontinuities, which are critical for stylized rendering.

Helmholtz stereopsis attempts to overcome some of these problems, seeZickler, “Helmholtz Stereopsis: Exploiting Reciprocity for SurfaceReconstruction,” ECCV, 2002. Other active methods such as shadow carvingcompute a tighter hull by observing shadows, see Savarese et al.,“Shadow Carving,” Proc. of the Int. Conf. on Computer Vision, 2001.

Therefore, it is desired to provide a camera and an image renderingmethod that can use multiple images acquired of a scene under differentillumination conditions to generate an output stylized image withenhanced or de-emphasized information.

SUMMARY OF THE INVENTION

The invention provides a camera and method for acquiring and renderingstylized images of scenes that pose problems for traditional cameras,e.g., scenes with low-contrast, geometrically complex scenes, and actionscenes. The images can be still images, or a sequence of images in avideo.

The camera acquires multiple images of a scene under differentillumination conditions. Image features are detected by combiningpixel-level information from these images. Texture details are reduced,and context is added to generate the stylized images.

The different illumination conditions can be obtained by controlledlighting, or by naturally changing illumination. When controlledlighting is used, an edge classification procedure is applied, and imagereconstruction is based on a manipulation of gradients fields. Withnaturally varying illumination, useful information extracted from a highquality image of a scene provides context or detail for another image,perhaps with a lower quality, of the same scene.

By using different sequences of filter operations, detected features canbe highlighted, and unnecessary details from multiple images can bereduced or combined in the reduced complexity output image. Theresulting images are simpler and easier to understand. For example, themethod can be used for illustrating complex mechanical parts, imagesimplification for compression, image resizing, non-photorealisticrendering, and video surveillance.

The invention extracts image features by observing changes in imagebrightness due to changes in illumination. The change in illuminationcan be controlled or uncontrolled and naturally changing. Undercontrolled illumination, the location of the shadows is used as a cuefor detecting geometric features such as silhouette edges.

Under natural illumination, the invention relies on greater changes inthe lighting, such as the sun during the day and artificial lightsduring the night, to merge local features and effectively enhance imagesby providing context.

The invention provides procedures for detecting and processing imagefeatures to deal with low-contrast scenes, geometrically complex scenes,and action scenes.

The method according to the invention provides a procedure for edgeclassification that separates silhouette edges from texture edges. Theappearance of edges can then be enhanced based on qualitative depthinformation, e.g., in-front-of or behind relationships of objects in thescene.

The invention also provides a procedure for reducing detail in texturedregions while preserving geometric features.

The invention can also merge multiple images while preserving usefulinformation to improve information density in an image. Using spatialand temporal gradients can also enhance videos.

The invention also provides a self-contained stylized imaging device,that is, a non-photorealistic (NPR) camera that can directly generateNPR images of a scene.

The images or videos can be rendered in many ways, e.g., imitatingartist-drawn or cartoon-like style with edge highlighting to make theimage easier to understand.

The invention can also be used for other applications such asillustrating complex mechanical parts, image simplification forcompression, photo-booths and video surveillance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of a non-photorealistic camera according tothe invention;

FIG. 1B is a block diagram of the camera of FIG. 1A with an alternativearrangement of flash units;

FIG. 1C is a block diagram of the camera of FIG. 1A with anotheralternative arrangement of flash units;

FIG. 2 is a flow diagram of a method for generating a stylized imageaccording to the invention;

FIG. 3A is a flow diagram of a method for detecting silhouette edges;

FIG. 3B is an image of a vase of flowers with silhouette edgeenhancement according to the invention;

FIG. 4 is a flow diagram of a method for reducing detail in texturedimages;

FIG. 5 is a flow diagram of a method for enhancing the quality of imagesacquired of a scene according to the invention; and

FIG. 6 is a flow diagram of a method for enhancing the quality of avideo acquired under uncontrolled illumination according to theinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

NPR Camera

FIG. 1A shows a digital camera 100 for generating non-photorealistic(NPR) images according to our invention. The camera 100 includes aplurality of flash units 101–104, and a single lens 105. The flash units101–104 are distributed around a center of projection (COP) 106 of thelens 105. For best results, the flash units are placed as close to theCOP as possible. Therefore, it makes sense to have the flash unitsmounted on a front surface 107 of the camera. The flash units can bedistributed evenly around the COP.

In an optimal configuration shown in FIG. 1B, the flash units 101–104are operated in left-right and top-bottom pairs. This allows pixels inimages 110–114 to be imaged for at least five lighting conditions, noshadows under ambient lighting and no flash, and bottom, top, left, andright shadows. Thus, neighboring pixels are shadowed at least in oneimage, and not shadowed in at least one other image.

This configuration also makes an epipolar traversal efficient. For theleft-right pair, the traversal can be approximated along horizontal scanlines. For the top-bottom pair, the traversal is along the verticaldirection. FIG. 1C shows an arrangement with three flash units.

The flash units can be triggered by optically coupled LEDs. The LEDs areturned on in a sequence, one by one, by a microcontroller to activatethe corresponding flash unit. The flash duration is about 4milliseconds. One image is acquired while illuminating the scene withone of the flash units.

The resolution of the camera is about 4 Megapixels, although lower andhigher resolutions are also possible.

As with all modern digital cameras, our camera also includes amicroprocessor 120, and memory 130. The microprocessor 120 is designedto perform the operations described herein, specifically a method 200for generating a stylized image, see FIG. 2. The memory 130 is used tostore the acquired images, other intermediate images in a linearizedform. The memory also stores data used by the methods and proceduresdescribed herein. The output stylized image 201 can also be stored inthe memory, for later down-loading, via a port 140, to an externalprocessor for viewing.

It should be noted that the camera 100 can take multiple images in rapidsuccession to produce a sequence of frames or a video. These can also bestylized as described herein. It should also be noted that the images110–114 can be acquired by using other techniques consistent with theinvention, for later processing in a stand-alone processor, for example,a desk-top system or a portable computing device.

Camera Operation

FIG. 2 shows our method 200 for generating the stylized image 201 of ascene. By stylized, we mean any non-photorealistic image with specificdetails such as edges enhanced, and other details such as texture andflat backgrounds reduced, abstracted or otherwise simplified asdescribed herein.

First, the set of images 110–114 is acquired 210 by the camera 100 ofFIG. 1, or some other camera and flash units that replicate theconfiguration shown. Each image is acquired under a different controlledlighting condition. For the configuration shown, one image is taken withambient lighting, i.e., no flash, one with the top flash unit, one withthe bottom flash unit, one with the left flash unit, and one with theright flash unit. Minimally, two flash units and three images arerequired, better results can be obtained with additional images andflash units, e.g., four to eight.

The set of images is processed by the microprocessor 120 to detect 220silhouette edges, and identify 230 texture regions. Then, the silhouetteedges and texture regions of a combination of the images are used toprovide the stylized image 201. For example, a width of silhouette edgesis made proportional to a depth discontinuity at the edges, and texturedregions are de-emphasized according to color gradients.

We classify each pixel in each image as being either a silhouette edgepixel, a texture edge pixel, or a featureless pixel.

We use the term silhouette edge pixels to refer to those pixels with aC0 depth discontinuity in the scene. These include pixels thatcorrespond to points on the surface of an object in the scene where thenormal is perpendicular to a viewing direction. We also include pixelscorresponding to boundaries of thin objects, e.g., a leaf or sheet ofpaper, and view-independent edges of objects such as a cube, both ofthese type of objects have depth discontinuities. Silhouette edge pixelscan also belong to the interior of an object due to self-occlusion.

Texture edge pixels correspond to changes in reflectance and materialdiscontinuities in the scene. Textured regions are typically delineatedby texture edges. However, texture regions may not be completelyenclosed.

Pixels in featureless regions correspond to regions in the scene withnear constant reflectance and low curvature, e.g., a flat backgroundwall. These pixels can also correspond to minor changes in appearancedue to changes in the illumination or viewing direction, includinganisotropic reflections on “shiny” materials such as metal and plastic.

Image Acquisition

The set of images 110–114 is acquired 210 of a scene with the flashunits 101–104 placed very close to the center of projection (COP) 106 ofthe lens 105, see FIG. 1. Due to the small baseline between the COP andthe flash units, a narrow sliver of a shadow appears near eachsilhouette edge in the image that presents depth discontinuities in thescene.

By combining information about attached cast shadows from two or moreimages with distinct lighting, we can detect the silhouette edge pixels.

Herein, an attached cast shadow is defined in an image space. Thisdefinition is quite different from the traditional shape-from-shadowdefinition in object space. There, the surface boundary, where a lightray is tangential to a smooth object, is considered to have attached or‘self’ shadows. Here, the attached cast shadow is generated when theobject and the shadow cast by the object on the background arecontiguous in image space.

For most view independent edges, the shadowed areas include shadows caston a next depth layer. For view-dependent edges, a small part of theshadow area includes self-shadows.

Generally, shadows are on the ‘opposite’ side of the flash units, i.e.,if the flash unit is to the right of the lens, then the shadows appearto the left of the depth discontinuities in the camera image.

Detecting Silhouette Edge Pixels

FIG. 3A shows a procedure 300 for detecting silhouette edge pixels thatis basic to our approach. The idea is surprisingly simple. Although ourimage acquisition process is closely related to photometric stereo, asfar as we know, it has not been used in computer vision systems fordetecting depth discontinuities as reflected by shadows in a scene. Itallows us to classify other edges by the process of elimination.

The basic procedure 300 operates as follows.

Acquire an ambient image I_(ambient) 301 of a scene, where I_(ambient)is an image taken under ambient lighting without any of the flash units.

Acquire n controlled illuminated images I′_(k) 302 of the scene, fork=1, . . . , n with a point light source, i.e., a flash unit, atposition P_(k). The images can be acquired in any order. In our example,n=4.

Difference images I_(k) 303 are obtained by subtracting 310 the ambientimage from the illuminated images, I_(k)′−I_(ambient).

A maximum image I_(max) 304 is generated 320 from difference images asI_(max)(x)=max_(k)(I_(k)(x)), k=1, . . . , n. That is, each pixel in themaximum image has a maximum intensity value from any of thecorresponding pixels in the difference images.

For each difference image I_(k), generate 330 a ratio image 305 bydivision, for all pixels (x), R_(k)(x)=I_(k)(x)/I_(max)(x). That is,each difference image is divided by the maximum image.

A pixel e_(k) 341 in the ratio image is an epipole pixel, i.e., theepipole pixel is an image of the corresponding light source at P_(k).Conventional stereoscopic techniques can be used to locate the epipole.

For each ratio image R_(k) 305, traverse 340 the image while comparingpixel intensities with the pixel intensity of the epipole pixel. Byradial traversal, we mean detecting intensity edges along a radial linefrom the epipole pixel to a given pixel. This traversal detectstransitions from lighted areas to shadow areas, and vise versa.

Locate pixels y with steps with a negative intensity transition. Markthese pixel y with the negative intensity transition as a silhouetteedge pixel 306. All pixels identified as silhouette pixels can beoverlaid in the maximum image, so that when all images have beenprocessed, the maximum image shows all silhouette contours.

FIG. 3B is an image 390 of a vase of flowers with silhouette edgeenhancement according to the invention.

With a sufficiently large n, minimally two but typically four to eight,silhouette edge pixels at all orientation and sufficient depthdifference can be detected. If the silhouette has a componentperpendicular to the epipolar line, then an edge pixel with negativeintensity transition from the illuminated the shadowed area is detected.

It should be understood, that for a very simple scene some silhouetteedges could be detected from a single illuminated image. For example, ina scene with mostly vertical depth discontinuities, e.g., a picketfence, edges can be detected by a single side illumination.

We now describe the procedure 300 in greater detail. The I_(max) imageis an approximation of the image with the light source at the camera'sCOP 106. This approximation is close when the point light sources havethe same magnitude, and a baseline between the light sources issubstantially smaller than a depth of the scene. Thus, the I_(max) imagehas no pixels in a shadow.

The ratio (I_(k)/I_(max)) is substantially close to 1 in areasilluminated by light source k, and substantially close to zero inshadowed areas. Hence, the locations of the negative intensitytransitions mark the shadow edges.

An intensity at a pixel x for a 3D scene point X with diffuse reflectionin the camera coordinate system, given light source at P_(k) is given byI_(k)(x).

If the 3D scene point X is illuminated by light source P_(k), thenI _(k)(x)=μ_(k)ρ_(k) L _(k)(x)•N(x), otherwise,I _(k)(x)=0,

where value μ_(k) is the magnitude of the light, ρ_(k)(x) is thereflectance at point X, L_(k)(x) is the normalized light vectorL_(k)(x)=P_(k)−X, and N(x) is the surface normal, all in the cameracoordinate system.

Thus, when X is illuminated by the point light source P_(k), the ratiois as follows:R _(k)(x)=I _(k)(x)/I _(max)(x)=μ_(k)(L _(k)(x)•N(x))/max_(i)(μ_(k)(L_(i)(x)•N(x)).

For diffuse objects with non-zero ρ_(k), R_(k)(x) is independent of thealbedo ρ_(k), and only a function of the local geometry. Furthermore, iflight sources are close to camera's COP 106, and X>>P(k), then thisratio is approximately (μ_(k)/max_(i)(μ_(k))). which is a constant for aset of omni-directional light sources. Note, R_(k)(x) is very low nearsilhouettes of a curved object with view dependent edges.

This is because (L_(k)(x)•N(x)˜=0), and the dot product for lightsources on the opposite side is larger, i.e., (L_(i)(x)•N(x)>L(x)•N(x)).Thus, the intensity of pixels in the ratio image R_(k)(x) decreasesrapidly even though the pixel is not in a shadowed area. However, thisis not a major issue and results in a thickening of the shadowed regionand does not lead to a reversal of intensity profile along the epipolarline.

Due to secondary scattering, the ratio R_(k)(x) is thus substantiallyclose to zero in shadowed areas. The intensity profile along theepipolar line shows a sharp negative transition at silhouette edge as wetraverse from non-shadowed foreground to shadowed background, and sharppositive transition as we traverse from shadowed to non-shadowed regionon the background.

Any standard 1-D edge detector can be applied along the radial epipolarline from the epipole to a given pixel detects both these transitions,and we mark pixels with the negative transition as a silhouette edgepixels.

Because we detect a transition, noise and secondary scattering canaffect the accuracy of the position of the detected silhouette edgepixels, but always their presence.

There are a few conditions when a negative transition at a silhouetteedge cannot be detected in the ratio image R(x), or when otherconditions cause spurious transitions. The silhouettes can be missed dueto detached shadows, low background albedo, holes and valleys, or whensilhouettes lie in shadowed regions. Some pixels can be misclassified assilhouette edge pixels due to specularities, self-shadows, orview-dependent silhouettes.

There is a tradeoff in selecting the baseline between the camera's COPand the flash units. For a larger image width for shadow d, a largerbaseline is better, but a smaller baseline avoids separation of shadows.

A width of a particular attached cast shadow in an image isd=f(z₂−z₁)B/(z_(1·z) ₂), where f is the focal length, B is the baselinein mm, and z₁ and z₂ are depths, in mm, to the shadowing and shadowededge.

The shadow separation occurs when a threshold width T of the object issmaller than (z₂−z₁)×B/z₂. Therefore, a smaller baseline allows asmaller width T without shadow separation. Because the effective focallength can be increased using a higher resolution camera, we use ahigh-resolution camera with a very small baseline and large distance or‘depth’ to the object (z₂).

Non-uniformity of lights sources can also affect the results. It affectsR_(k)(x) because (μ_(k)/max_(i)(μ_(k))) is not a constant. Fortunately,even with non-uniform light sources, the lobe intensity changes smoothlyacross the field, so it does not introduce spurious edges in R_(k).

Detection of negative intensity steps leading into a shadow region is abinary decision, making our method robust to noise. However, lightproduced by the flash units still needs to override the ambient light.We also rely on the assumption that silhouettes are separated from thebackground with a finite depth, which implies we need a background.

Reducing Detail in Textured Regions

We also provide a procedure for reducing details or complexity inregions of images that are not associated with the silhouettes of ascene, such as texture and illumination variations. Given thesilhouettes edge pixels 306, it is now possible to identify 230 pixelsbelonging to texture edges. Thus, textured regions can be identified.These pixels are independent of the direction of illumination. Textureedges are intensity edges in the maximum image I_(max) minus thedetected silhouette edges.

Ideally, we would like to identify all textured regions, i.e., sets ofpixels corresponding to a texture detail in the original image, andde-emphasize the texture detail in the output image 201. However,although we can identify texture edges, we cannot reliably find alltextured regions. This is because texture edges do not always form anenclosed contour around a textured region, due to gaps after silhouetteedge detection, or because such regions can fade into dominant colors.

Therefore, we desire to perform a ‘tunable’ abstraction where, forexample, geometric features can be given more importance, and texturefeatures, e.g., edges and regions, can be de-emphasized. We also want toeliminate pixels that do not have the dominant colors.

One solution blurs regions associated with the texture edges. One couldsimply assign a weighted average of the colors of the neighboringfeatureless pixels to texture pixels and nearby pixels. However, thatonly diminishes the textured region boundaries and does not remove thegradients. Furthermore, the textured regions, delineated by textureedges, can be several pixels wide, and thus, are not completelyeliminated. Another approach uses distance fields or diffusion typetechniques.

Instead of pixel-level operations, we use an edge-based procedure.

Our edge-based procedure is based on the observation that high intensitygradients at texture pixels separate non-dominant color pixels from thedominant color pixels. If the image is reconstructed from gradientswithout the high gradients at texture pixels, then the non-dominantcolor pixels are automatically filled in smoothly by colors from theother side of the texture pixels. No decision needs to be made aboutwhat intensity values to use to fill in holes, and no feathering orblurring needs to be done, as is required with conventional pixel-basedsystems.

FIG. 4 shows the steps of our rendering procedure 400. First, generate410 a mask image M(x, y) 411 from the acquired images 1101–114. Astarting point can be the maximum image overlaid with all the silhouetteedge pixels. An intensity of a pixel (x, y) in the mask image M(x, y)

-   -   =0, if (x, y) is a texture edge pixel    -   =d(x, y), if (x, y) is a featureless pixel    -   =1, if (x, y) is a silhouette edge pixel.

The factor d(x, y) is a ratio of a distance field of the texture edgepixel and a distance field of the silhouette edge pixel.

Then, determine 420 intensity gradients ∇I(x, y) 421 in the mask image.Next, integrate 430 the gradients in the masked image according toG(x, y)=∇I(x, y)·M(x, y)431.

Construct 440 an image I′(x, y) 441 that minimizes |∇I′−G|, andnormalize 450 colors in image I′(x, y) to substantially match the colorsin the input image I(x, y).

For image reconstruction 440 from gradients fields, an approximateinvertibility problem, we adapt a direct method described by Elder, “AreEdges Incomplete?” International Journal of Computer Vision 34, 2/3, pp.97–122, 1999, and Fattal et al., “Gradient Domain High Dynamic RangeCompression,” Proceedings of Siggraph, ACM SIGGRAPH, 2002.

An estimate of the intensity function I′, so that G=∇I, can be obtainedby solving the Poisson differential equation ∇I²=divG with a LaPlaceoperator and a divergence operator.

We use a full multigrid method to solve the LaPlace equation, see Presset al., “Numerical Recipes in C: The Art of Scientific Computing,Pearson Education, 1992.

At the border of the image, we specify boundary conditions to reduceimage artifacts near the boundary. A natural choice is the Neumanncondition ∇I.n=0, i.e., the derivative in the direction normal to theboundary is zero. This is clearly not true when high gradients arepresent near the image boundary. Therefore, we modify the image I′, andpad it with colors obtained by applying a Gaussian filter to smoothboundary pixels. The constructed image I′ is later cropped to itsoriginal size. Padding the border by five pixels is sufficient. Finally,we normalize 450 the image to obtain the color values in the reduceddetail output image 401.

We assign and modify not just the luminance, but also texture.Therefore, we integrate 430 the masked gradients separately in eachcolor channel. Without a consistent boundary condition (∇I.n=0), thescale and shift in the normalization would be significantly different.

Rendering

Color assignment

Image construction 440 from the gradient field involves a scale andshift ambiguity.I″(x, y)=c ₁ I′(x, y)+c ₂.

We compute the unknowns, c₁ and c₂, in the least square sense, using asimple heuristics that demands that the overall appearance of thereconstructed image substantially matches the original image. Each pixelleads to a linear equation,I(x, y)=c₁ I′(x, y)+c ₂.

If we operate on purely the luminance image and add back thechrominance, then the textures are not removed. In fact, there is nocolor space in which textures can be completely separated. Hence, we doimage construction in all three color channels separately and determinethe unknowns per channel.

Because of the boundary condition (∇I.n=0), the scale and shift in thethree channels do not introduce noticeable artifacts.

Edge Enhancement

Our procedure to detect silhouette edge pixels also generates additionaluseful information that can be used for image enhancement.

The width of a shadow d=f(z₂−z₁)B/(z₁z₂), is proportional to a depthdifference (z₂−z₁) at the depth discontinuity. This information can beused during image enhancement.

At a silhouette edge, we can determine which side of the silhouettebelongs to the foreground and which side belongs to the background. Thecalculation is based on which side of the edge has a higher intensity inthe R_(k) image at the negative transition along the epipolar line.

This qualitative depth relationship can also be used for edgeenhancement. We first generate a silhouette image, where silhouettes arein white on a black background. We convolve with a filter that is thegradient of an edge enhancement filter. We use a Gaussian filter minusan impulse function. When we integrate the convolved image, we obtainsharp transition at the silhouette edges.

Comparison with the Prior Art

Better lighting could be used to improve contrast and highlight objectswith respect to the background. While the success of this approachgenerally depends on the skill of the photographer, we provide a generalsolution with clear geometric basis. In addition, because the lightsources are close to the camera's COP, our single self-sufficient camera100 does not require external light sources, simplifying the set up.

A second simple option is to perform edge-detection on the intensityimage. However, sharp changes in image values do not necessarily implyobject boundaries, and vice versa. For example, a complex scene cangenerate many spurious intensity edges, while in a simple image very fewintensity edges are detected. Hence, image enhancement strategies thatrely on examining image value changes alone often fail. Previous NPRtechniques that work on a single image rely on very high quality, goodcontrast images so that intensity edges detection and color segmentationare reliable.

Our approach may appear to be similar to active illumination techniques,such as traditional stereo, photometric stereo and Helmholtz stereo.However, depth discontinuities present difficulties for a traditionalstereo technique. Stereo techniques often fails due to half-occlusions,which confuse the matching process.

Photometric stereo simultaneously estimates geometry and albedo across ascene. The main limitation of classical photometric stereo is that thelight sources must be far apart from the camera's COP, and the positionsof the light sources must accurately known. This requires a fixedlighting rig as might be possible in an studio, industrial or laboratorysetting, but not in a self-contained camera unit. Furthermore, thatapproach relies on detecting normals along smooth surfaces, and fails atdepth discontinuities, shadowed regions and half-occluded sections. Ourapproach is exactly opposite, and takes binary decisions for intensityvariations at a scene depth discontinuity.

For Helmholtz stereo, shadowed and half-occluded regions are incorrespondence. A surface in a shadow in a left image is not visible inthe right image, and vice versa. The problem is that computing shadowedregion in a single image is a difficult problem. The only reliable wayto classify a pixel to be in shadow region is to compare it with yetanother pixel when it is not in the shadow. This binary decision makesour technique robust. We also do not rely on any matching or correlationprocess, and most operations can be done on a per pixel basis. Thisenables us to integrate the entire method in the camera'smicroprocessor.

Enhancing Images of Natural Uncontrolled Illuminated Scenes

In addition to making use of controlled illumination, we can alsoenhance images taken of scenes with naturally changing illumination. Bynaturally uncontrolled illumination, we mean that both the direction andintensity of illumination is uncontrolled.

It is often difficult to interpret a low quality image due to dark orhazy areas, e.g., in nighttime or foggy scenes, low-resolution, or noisyimages. Our idea is to enhance these low quality images by providingcontext using other high quality images acquired of the scene withbetter illumination, e.g., under daylight. Providing context can beuseful for both static and dynamic scenes.

Static Scenes

When the geometry of a scene remains constant but for the changes inillumination, the context can clarify the scene and identify areas ofinterest. For example, a low quality nighttime image of a scene can beenhanced by combining it with a high quality daytime image of the samescene.

Dynamic Scenes

More interesting applications are when there is a change in scenegeometry over time. Using a notion of a static background and a dynamicforeground, we can provide a context of an action or event. The dynamicscene component can be captured in still images or a video.

Some example applications where our image enhancement method can be usedinclude surveillance, e.g., determining the part of a building nearwhere a person is standing, determining what a person's hand is hiding,identifying reflections in dark areas, e.g., headlights reflecting fromwindows of dark buildings, or identifying a blinking light at night.

There are two possible types of enhancements, symmetric and biased. Inthe symmetric enhancement, all images are treated as having the samequality, and all “interesting” pixels from all images are used duringthe enhancement process. In the biased enhancement, the high qualityreference image that is used to improve the quality of the low qualityimages is identified explicitly, and more importance in the lowerquality images are retained even when more important features appear inthe high quality reference image.

The problem reduces to how best to combine important information fromtwo or more images. Our approach is based on two heuristics. First weretain pixels from each image that appear to be locally importantdepending on the specified bias. Second, to maintain intra-imagecoherence, we provide context to pixels in the images. Note, we do notimprove the quality of individual pixels themselves, but simply givesufficient context to improve human interpretation of the images.

Hence, any conventional techniques, such as contrast enhancement,histogram equalization, mixed Gaussian models for background estimationare orthogonal to our approach, see Toyama et al., “Wallflower:Principles and Practice of Background Maintenance,” ICCV (1), pp.255–261, 1999. However, those techniques could be applied in a postprocessing step to further improve the final result.

We build our results on the basic observation that if the camera andviewed geometry remain static, only illumination and minor parts of thescene change, e.g., moving subjects like people, devices, vehicles, soit is possible to collect information over time.

FIG. 5 shows the steps of our method 500 for enhancing the quality ofimages acquired of a scene. First images I_(n) 501, for I=1, . . . , nare acquired 510 of the scene. The images are acquired with differentintensities and directions of illumination so that some images have ahigh quality, and other images have a low quality. For example, some ofthe images are taken under daylight, while other images are taken atnight.

For each input image, determine 520 the intensity gradients 521 of thepixels according to G_(i)=∇I_(i), for I=1, . . . , n.

Next, weights W_(i)(x, y) 531 are determined 530 for each pixel in eachimage. The weights correspond to the importance of various portions ofthe images. The weights are multiplied by the gradients and summed 540over all images according to G(x, y)=Σ_(i)W_(i)G_(i)(x, y) to determinea gradient field G. Then, an enhanced image 1 551 is constructed 550from the gradient field G, and the pixel intensities in the enhancedimage 551 are normalize 560 to obtain the final output normalizedenhanced image 509.

The weights W_(i)(x, y) 531 can be determined in several ways dependingon the application. Our basic idea relies on the widely acceptedassumptions that the human visual system is not very sensitive toabsolute luminance reaching the retina, but rather responds to changesin ratios of local intensities.

Therefore, our importance weights are based on spatial intensitygradients in images. For a video, temporal intensity gradients can alsobe considered. We first generate a weighting image to encode theimportance that combines the gradients from day and night images. Thisis a mixed-gradient image. By integrating this mixed-gradient image, wecan construct the enhanced image that preserves local importance in bothimages. Because gradient field integration involves computing a smoothintegral, we automatically get a gradual transition between the sourceimages.

The weights W_(i)(x, y) encode the importance of a pixel in an image andensure intra-frame coherence. Let us consider some symmetric approaches.A binary 0 or 1 weight W_(i)(x, y) can be assigned by checking whichimage has the maximum variance at pixel (x, y). Normalized value in therange [0,1] can be assigned, based on the weighted average of thevariance of the source images.

For intra-frame coherence, we multiply by a weighted distance field of abinary image obtained by thresholding high variance regions. For biasedapproaches the weights are scaled by a scalar B_(i), i.e., the bias ofeach image.

We provide a practical method for improving a low-quality night-timeimage by combining it with a high-quality day-time image of a scene.This idea appears to be very simple in retrospect. So far, this idea hasnot been used for non-photorealistic image synthesis.

A naïve approach to automatically combining a daytime and nighttimepicture would be to use a pure pixel substitution method based on someimportance measure. This works well only when the source images arealmost identical, e.g., two images of the same scene with differentfocus.

Similarly, blending strategies such as max_(i)(I_(i)(x, y)) oraverage_(i)(I_(i)(x, y)) also cause problems. For example, whencombining day and night images, one needs to deal with the high variancein the day-time images and with mostly low contrast and patches of highcontrast in night-time images. Taking the average simply overwhelms thesubtle details in the nighttime image, and presents ‘ghosting’ artifactsaround areas that are bright at nighttime. Furthermore, juxtaposing orblending pixels usually leads to visible artifacts, e.g., sudden jumpsfrom dark night pixels to bright day pixels, which distract from thesubtle information conveyed in the night images.

Low-Quality Video Enhancement

Providing context to captured events and actions can enhance low qualityvideos. The context, as described above, comes from higher qualityimages. Videos, however, present several additional challenges:inter-frame coherence must be maintained, i.e., the weights insuccessive images should vary smoothly; and a pixel from a low qualityimage may be important even where the local variance is small, e.g., thearea between the headlights and the taillights of a moving car.

Our solution is based on the simple observation that moving objects spanapproximately the same pixels from front to back, with respect to thedirection of travel. For example, the front profile of a moving carcovers all the pixels that will be covered by rest of the car insubsequent frames, until the back of the car is reached. Although themoving object may not show enough intra-frame or inter-frame variance,we maintain the importance weight high for a large interval after thefront of the object, using temporal hysteresis.

The importance, based on the spatial and temporal variation and thehysteresis at a pixel is encoded in weights W^(j)(x, y) for each framej.

Just as in the case for static images, a good quality video segmentationor optical flow technique could improve our results. Here, weintentionally use a very simple technique, i.e., pixel-wise differences,to demonstrate that our technique does not need to rely completely oncomplicated optical flow or image change detection techniques.

FIG. 6 shows the steps of our method 600 for enhancing the quality of avideo acquired under uncontrolled illumination. First, acquire 610 aninput video V^(J) 611 of a scene under uncontrolled illumination. Next,smooth 620 the input video into a noise-reduced video S^(j) 621, anddetermine 630 gradients G_(t) ^(j) (x, y) 631 for each frame j of thesmoothed video. A binary mask M^(j)(x, y) 641 is determined 640 bythresholding the gradients G_(t) ^(j) (x, y) 631, and weights 651 aredetermined 650 for the binary mask according toW ^(j)(x, y)=F(M ^(k)(x, y), k=j−c:j+c,

where c is a temporal extent, e.g., ten frames in each direction.

Then, for each frame j, if W^(j)(x, y)>0, a gradient field is computed660 asG _(xy) ^(j)(x, y)=W ^(j)(x, y)G ^(j) _(xy)(x, y)+(1−W(x, y))G* _(xy)(x,y), elseG _(xy) ^(j)(x, y)=(1−W(x, y))max(S _(xy) ^(j)(x, y), G* _(xy)(x, y)),

where G*_(xy) 662 are the gradients in a single high-quality imageacquired from the scene under ‘ideal’ illumination.

Now, frames of the output video V^(j) _(out) 609 can be constructed 670from the gradient field G_(xy) ^(j).

The noise reduced video S is generated by applying edge-preservingbilateral filtering in space and time to the frames of the input video.

The temporal gradient at a pixel (x, y) is G^(j) _(t)(x, y)=S^(j)(x,y)−S^(j−1)(x, y), i.e., a difference between a current and previousframe of the noise-reduced video.

The spatial gradient G_(xy) ^(i)(x, y)=(V^(j)(x, y)−V^(j−1)(x−1, y),V^(j)(x, y)−V^(j−1)(x, y−1)) is calculated from the original video tomaintain the detail.

A weight W^(j)(x, y) is computed by applying a distance field to thebinary mask values {M^(k)(x, y)}, k=j−c: j+c. This achieves temporalcoherence for assigned weights. We select the extent of influence, c, tobe ten frames in each direction.

Finally, we use the computed weights to combine the spatial gradientsfrom each frame with the spatial gradients, G*^(xy) of a high qualityimage I*. If the weight due to temporal variation is zero, then werevert back to pure image gradient comparison. Note, we use the gradientfrom a smoothed frame S^(i) to avoid accentuating intra-frame noise.

For a practical implementation we used a 3D Susan filter, i.e., a 3×3×5neighborhood, with σ=15 and τ=20. The high-quality static image isobtained by median filtering a day time video.

Other Applications

The edge classification procedure according to the invention can alsoimprove edge-based and area-based stereo correspondences. Matching ofedges is improved when silhouette edges are properly matched. Area-basedtechniques work poorly due to partial occlusion, but such problematicareas can be detected. Edge classification can also assist color andtexture segmentation techniques even in low-contrast images, which inturn can be used for layer extraction and image compression.

The silhouette edge detection can also be used in computer vision andimage processing applications such as stereo reconstruction, visual hullconstruction, face recognition, and an initial step for technicalillustration and cartoons.

It should be noted that our silhouette extraction scheme can also beused for other images that have similar ‘shadow’ problems, for example,images taken with infrared cameras, laser range scanners, and radar.

Our video enhancement scheme is ideal for video surveillanceapplication. Sophisticated prior art schemes try to analyze the image inorder to detect and display significant events. In contrast, ourapproach is simple. We provide the viewer with tunable density ofinformation.

Effect of the Invention

The invention provides novel techniques for extracting usefulinformation from multiple images of scenes by exploiting theirillumination-dependent properties. The invention has been described interms of basic prototypes, related procedures for edge enhancement,image processing and video processing.

By removing extraneous details, the invention can clarify ageometrically complex or low contrast scene. By providing context todark or low quality images or videos, the invention can generate moreuseful images and better video surveillance. Our techniques can easilybe implemented in software.

Alternatively, the methods described herein can be implemented withhardware inside a single camera. By performing ‘one-pixel-at-a-time’processing on acquired images, the camera can directly output stylizedimages.

Although the invention has been described by way of examples ofpreferred embodiments, it is to be understood that various otheradaptations and modifications can be made within the spirit and scope ofthe invention. Therefore, it is the object of the appended claims tocover all such variations and modifications as come within the truespirit and scope of the invention.

1. A method for enhancing images of a naturally illuminated scene,comprising: acquiring a plurality of images I_(i) (x, y) of a scene,each image having a different uncontrolled illumination; determining,for each image I_(i) (x, y), intensity gradients according to Gi(X,Y)=∇I_(i)(x, y); determining, for each image li(X, Y), weights Wi(X, Y);determining a gradient field G(X, Y) from a weighted sum of the weightsmultiplied by the intensity gradients according to G(X, Y)=E_(i)WiGi(X,Y); constructing an enhanced image I′ from the gradient field; andnormalizing pixel intensities in the enhanced image.
 2. The method ofclaim 1 wherein the scene is static.
 3. The method of claim 1 whereinthe scene is dynamic.
 4. The method of claim 1 wherein intensities anddirections of the illumination are uncontrolled.
 5. The method of claim1 wherein the weights are based on spatial intensity gradients.
 6. Themethod of claim 5 wherein the weights are based on temporal intensitygradients.
 7. The method of claim 1 wherein each weight is either zeroor one.
 8. The method of claim 1 wherein the weighs are normalized in arange zero to one.
 9. The method of claim 1 wherein a first image of thescene is acquired during daytime, and a second image of the scene isacquired during nighttime.
 10. The method of claim 9 further comprising:weighting the first image according to intensity gradients of the firstimage; and weighting the second image according to intensity gradientsof the second image.
 11. The method of claim 10 wherein weights for theweighting are based on spatial intensity gradients.
 12. The method ofclaim 10 wherein weights for the weighting are based on temporalintensity gradients.
 13. The method of claim 10, further comprising:normalizing weight for the weighting.
 14. The method of claim 9, furthercomprising: normalizing pixel intensifies in the enhanced image.
 15. Themethod of claim 1 wherein a direction and intensity of the illuminationis uncontrolled.
 16. The method of claim 1 wherein the differentillumination for each image is due to weather conditions.
 17. The methodof claim 1 wherein the different illumination for each image is due to atime of day.
 18. The method of claim 1 wherein the combining treats theplurality of images equally.
 19. The method of claim 1 wherein thecombining treats the plurality of images unequally.
 20. The method ofclaim 19 wherein a first image is a low resolution of the scene, and asecond image is a high resolution image of the scene, and the combiningenhances a quality of the first image.
 21. The method of claim 1 whereinthe combining further comprises: retaining, for each image, only pixelsappearing locally important based on a specific bias.
 22. The method ofclaim 1 wherein the combining further comprises: maintaining anintra-image coherence to provide a context to pixels in the enhancedimage.
 23. The method of claim 1 wherein the plurality of images areacquired with different focuses.
 24. The method of claim 1 wherein theillumination is infrared radiation.
 25. The method of claim 1 whereinthe illumination is laser radiation.
 26. The method of claim 1 whereinthe illumination is radar.